A typical digital camera uses a single color image sensor to capture a representation of a scene, including a representation of the color in the scene. The colors are separated by a color filter array (CFA) that overlies the sensor and associates each image pixel with a particular color according to the particular color pattern of the CFA. For example, the Bayer CFA (described in U.S. Pat. No. 3,971,065) provides an RGB pattern with a checkerboard arrangement of green pixel sites. Therefore, the primary CFA image signal for each pixel relates to but one color, that is, to either red, green or blue (for a Bayer CFA). After the CFA image is captured, the remaining (missing) colors for each pixel site are interpolated for each pixel from, e.g., surrounding pixels, so that a full resolution interpolated record is generated for each image. Each pixel in the interpolated record therefore has a set of RGB values.
For many years, some types of digital cameras, such as the Kodak DC50 camera introduced in 1995, have stored the CFA image data as a “raw” TIFF image file on a removable memory card. Such a file is called a “raw” file because significant image processing, including CFA interpolation, is required before the image can be displayed or printed. The well-known TIFF (Tag Image File Format) allows different types of image data, including CFA image data, to be stored using a standard image file wrapper. ISO 12234-2:2001, Electronic still-picture imaging—Removable memory—Part 2: TIFF/EP image data format standardizes the way in which CFA image data, and metadata tags that specify the color arrangement of the CFA, can be stored within a TIFF file. However, before such a “raw” file can be displayed or printed, it must be interpolated and converted to standard color image data, such as sRGB color image data. This standard color image data can be displayed or printed, or can be compressed and stored using a standard file format, such as for example a JPEG file format, a JPEG 2000 file format, or a FlashPix file format.
The process used to convert the image to a standard file format, and to print and edit the standard format, is shown in FIG. 1. The CFA image is captured in a capture step 10 by the camera. When the camera or memory is connected to a host computer in a connect step 12, the images are downloaded from the camera or memory card in a downloading step 14, processed in a processing step 16 and stored in a standard file format in a file formatting step 18. In the processing step 16, the CFA images are CFA interpolated as they are imported, using known CFA interpolation, color correction, and sharpening techniques, in order to create a “finished” RGB image file having a standard size. The finished RGB image file thus contains the CFA interpolated image, which has the same number of total pixels as the original image. Unlike the original image, however, the CFA interpolated image has complete RGB data for each pixel.
The advantage of using a standard file format is that it allows the image to be used by many different image software programs, photo printers, retail photo kiosks, and Internet based printing services supplied by many different companies. For this reason, most current digital cameras include a mode that produces and stores standard image files, such as JPEG image files. These image files are produced in the digital camera as the images are captured.
Many different standard image file formats are known and useful with this invention. One example of such a standard image file is a FlashPix file. FIG. 2 is a simplified diagram showing the key information stored in a FlashPix file. The FlashPix image format (defined in FlashPix Format Specification, version 1.1, (Digital Imaging Group, Jul. 10, 1997)) has been developed to serve as both an “interchange” format between devices (e.g. cameras) and applications (e.g. computer picture editing packages), and as a “native” format for image editing that allows the images to be easily and rapidly edited. This is accomplished using a hierarchical, tiled image representation within a “structured storage” file. Referring to FIG. 2, a FlashPix file contains the complete image data 24 plus a hierarchy of several lower-resolution copies within the same file (one set of lower resolution image data 25 is shown in FIG. 2).
Images at each resolution also are divided into rectangular tiles (e.g., squares), which enable the application to minimize the amount of image data processed to access, display or print a portion of the scene content. FlashPix allows thumbnail image data 23 and optional “ancillary” property set data 21 and 22 to be stored in the same structured storage file, along with the image data. This ancillary data can include digital audio recordings and parameters such as the time and date the photo was taken, the camera zoom position and focus distance, the scene illumination level, camera calibration data, image copyright owner, etc. For further information about the FlashPix image format see the aforementioned FlashPix Format Specification, version 1.1, (Digital Imaging Group, Jul. 10, 1997), which is available on the Wide World Web at http://www.i3a.org, and is incorporated herein by reference.
The image processing used to “finish” the image prior to storing the finished data in the FlashPix file includes operations such as CFA interpolation, color correction, and image sharpening. The output images are normally stored in a standard color space, such as the sRGB space which is supported by the FlashPix image format. As a consequence of generating the image file, the original camera data is discarded (step 26). If an image is to be printed or transmitted in an application step 28, the standard file format is first edited in an editing step 27.
To create final printed images, the finished image file may be adjusted by the user via an image processing program to create a final output image of the desired size, which may include only a “cropped”, color adjusted portion of the image captured by the camera, or may, e.g., include multiple images in a “montage”. More specifically, to provide the appropriately sized output image, the image processing program first crops the 750×500 RGB CFA interpolated image data, and then uses another interpolation algorithm in the editing step 27 to convert from the cropped version of the “finished” RGB image file to the final desired output image.
Note that this prior art approach uses two interpolation steps, one (step 16) to interpolate “missing” RGB data from the image sensor while maintaining the 750×500 pixel data array, and a second (step 27) to interpolate from the selected crop to the pixel data array that provides the desired output size. The user therefore selects the final print size (for example 8″×10″), and the image is interpolated up to produce an image data record of appropriate size for the printer. For example, if the output device printed at 200 pixels per inch, the image data would be interpolated to 1600×2000 pixels by the computer or the printer, (and optionally sharpened) to produce the desired image size. In addition, this known approach often uses two separate sharpening operations, one on the 750×500 pixel CFA interpolated image data following the CFA interpolation step 16, and a second one on the interpolated 1 600×2000 pixel image data array just prior to printing.
The prior art approach has a number of problems. First, the process of converting the image from the CFA image data (which may for example produce 12 bit per color data) to the 8 bit per color (24 bit per pixel) sRGB color space may limit the quality of certain images, by reducing the color gamut and dynamic range of the captured image. Second, the two interpolation steps (the CFA interpolation that produces a standard size image and the interpolation in the computer or printer that produces the desired print size) provide more interpolation artifacts than would be produced using a single interpolation step that interpolates from the sensor CFA data directly to the output image data sent to the printer. Finally, using two separate sharpening steps can also produce artifacts.
What is needed is a digital camera and a digital imaging system that maintains the advantages of using a standard, “finished” image file format so that the images can be used by many applications, yet also enables certain types of devices, including image editing software, to access the raw camera data in order to provide improved image quality when editing and printing the images.